Transformer‑based models

scGPT: Masked Gene Modeling

BERT‑style · cell × gene matrix

STEP

Cell × Gene Expression Matrix

cells as rows · genes as columns · values = log₁p(count+1)

expressed [MASK] / predicted attention

scGPT (Wang et al., Nature Methods 2024) pre-trains on 33M cells. Each gene is a token: a gene ID embedding + expression value embedding. During pretraining, ~15 % of gene expression values per cell are randomly masked. The transformer attends bidirectionally across all unmasked genes and must reconstruct the masked values — the BERT recipe applied to single-cell biology. Click to pause.